fix(observability): report the miner's real input/output token split to PostHog - #10199
Conversation
…to PostHog Both engine coding-agent drivers read input and output tokens separately and then returned only their sum, so CodingAgentDriverResult carried a single blended tokensUsed. The miner's $ai_generation capture had nothing real to put in $ai_input_tokens/$ai_output_tokens and hardcoded them to 0, emitting the true figure under a non-standard tokens_used property that PostHog's own cost views do not read -- so miner spend registered there as zero tokens across the board. Carry inputTokens/outputTokens on the driver result, populated by both drivers from values they already had, and report them. The split is still never fabricated: a CLI that reports only total_tokens leaves it absent and the blended figure keeps riding in tokens_used, and a side that is missing or out-of-contract (negative/NaN/Infinity) stays absent rather than becoming a 0 that is indistinguishable from a real 0 once aggregated. Closes #10198
|
Tip ✅ LoopOver review result - approve/merge recommendedReview updated: 2026-07-31 12:37:01 UTC
Review summary Nits — 5 non-blocking
Decision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionAddressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #10199 +/- ##
=======================================
Coverage 92.21% 92.22%
=======================================
Files 934 934
Lines 114178 114210 +32
Branches 27593 27599 +6
=======================================
+ Hits 105294 105326 +32
Misses 7582 7582
Partials 1302 1302
Flags with carried forward coverage won't be shown. Click here to find out more.
|
![loopover-orb[bot]](https://avatars.githubusercontent.com/in/4139483?s=60&v=4){kind=small}
…nt (#10212) SECRET_KEY matches /token/i, so scrubRecord -- wired as posthog-node's before_send -- rewrote PostHog's own $ai_input_tokens and $ai_output_tokens to the "[redacted]" STRING, which PostHog then coerced to null on its numerically-typed properties. The result: not one AI call in the project has ever carried a token count. posthog.ai_events.input_tokens/output_tokens/total_tokens are NULL for every model over the retention window, including claude-sonnet-5 at 2,321 calls and $498.43 of real spend. $ai_total_cost_usd came through untouched because it has no secret-shaped word in it. PostHog derives $ai_input_cost_usd/$ai_output_cost_usd from tokens, so those could not be computed either -- and the miner-side split landed in #10199 would have been scrubbed the same way. A secret-shaped key holding a NUMBER is a counter, not a credential: every secret this module exists to catch is a string, and there is no numeric form of one to leak. Skip redaction for numbers only; a string, object, array or boolean under the same key is still redacted exactly as before. Deliberately general rather than an allowlist of the two $ai_* keys -- an allowlist goes stale the moment PostHog adds $ai_cache_read_input_tokens, and it would fail the same silent way. Closes #10211
1 participant
Summary
The miner reported 0 input tokens and 0 output tokens on every
$ai_generation, so PostHog's own LLM cost views were blind to AMS spend entirely. The real figure was emitted, but under a non-standardtokens_usedproperty those views do not read.captureMinerPostHogAiGenerationhardcoded the two properties, and its doc comment justified that as honest — "there is no input/output split available at this layer". That was true of that layer, but the layer below threw the split away: both engine drivers read the two sides and returned only their sum.agent-sdk-driver.tsreadusage.input_tokens/usage.output_tokens, then returned(inputTokens ?? 0) + (outputTokens ?? 0).cli-subprocess-driver.tsdid the same withusage.inputTokens/usage.outputTokens.So the fix is not to relabel anything — it is to stop discarding data that was already in hand.
CodingAgentDriverResultnow carriesinputTokens/outputTokensalongside the existing blendedtokensUsed, both drivers populate them from values they already read, and the miner reports them.The never-fabricate convention
costUsdandtokensUsedalready follow is preserved throughout:total_tokensgenuinely has no split; it leaves both sides absent and the blended figure keeps riding intokens_used. Deriving a split from a total would be an invention.NaN/Infinity), stays absent rather than becoming a0— once aggregated, a fabricated 0 is indistinguishable from a real one.0therefore survives as the miner-side fallback only where it means "no split known".Each driver's token fields are produced by a single expression and spread into every return site, so a driver cannot report a split that disagrees with its own blended total.
Closes #10198
Scope
type(scope): short summaryConventional Commit format, for examplefix(api): restore profile access checks.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Closes #123) — a linked open issue is required for every contributor PR.Validation
git diff --checknpm run actionlintnpm run typechecknpm run test:coveragelocally;codecov/patchrequires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.npm run test:workersnpm run build:mcpnpm run test:mcp-packnpm run ui:openapi:checknpm run ui:lintnpm run ui:typechecknpm run ui:buildnpm audit --audit-level=moderateIf any required check was skipped, explain why:
npm run build:minerand the@loopover/engineworkspace suite (942 tests) were both run and are green; the engine package was rebuilt before every test run so thedist/-importing engine tests exercised the new code rather than a stale build.test:coveragerun: 100% of the changed lines AND branches in all five, verified line-by-line against the lcov report rather than read off a summary percentage. Engine behaviour is covered frompackages/loopover-engine/test/**as well as the root suite, so the engine upload credits it independently.Tests added
Two existing tests asserted the hardcoded zeros and the "no fabricated split" rationale; both were updated to the corrected expectation rather than deleted, since the rationale itself is what changed.
packages/loopover-engine/test/agent-sdk-driver.test.ts): the split alongside the blended total; the split riding the failure results too, exactly liketokensUsed/costUsd(the session was billed either way); and a side left absent rather than zeroed when it is missing, out-of-contract, or whenusageis absent entirely.total_tokenskept as the blended figure without inventing a split from it;total_tokensplus both sides all reported together (the CLI's own total still wins over the sum, unchanged); and a single reported side left absent on the other.$ai_input_tokens/$ai_output_tokens, and the 0-fallback for a driver that only knows a blended total, including a partially-reported case.withCodingAgentAiGenerationCapture: forwards cost, blended tokens and the split verbatim, and leaves the split at 0 for a driver that reports only a total.Safety
UI Evidencesection below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.Token counts are metadata, not content: no prompt, diff, or transcript text is added to any event by this change, and the existing assertions that
$ai_input/$ai_output_choicesare never present still hold.UI Evidence
Not applicable — no visible UI, frontend, docs, or extension change.
Notes
The blended
tokens_usedproperty is deliberately kept rather than replaced. It is the only figure available for a provider that reports no split, and dropping it would lose data for exactly the callers that have the least of it.Two AMS surfaces remain uninstrumented and are out of scope here —
runChatGrounding(packages/loopover-engine/src/miner/chat-grounding.ts) and therunCodingAgentAttemptpath, which callscreateCodingAgentDriverdirectly and so bypassesconstructProductionCodingAgentDriver's capture wrapper entirely. Both warrant their own issue and change.